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DETAILED ACTION 



1. 



This action is responsive to communications of amendment received 6/22/2005. 



2. 



The disposition of the claims is as follows: claims 1-39 are pending in the application. 



Claims 1, 6, 1 1, 16, 20, 24 and 28-39 are independent claims. 



Claim Rejections -35 USC §102 



3. 



The following is a quotation of the appropriate paragraphs of 35 U.S. C. 102 that form the 



basis for the rejections under this section made in this Office action: 

A person shall be entitled to a patent unless - 

(a) the invention was known or used by others in this country, or patented or described in a printed publication in this 
or a foreign country, before the invention thereof by the applicant for a patent. 

4. Claims 1-10, 28, 28, 34, 35 are rejected under 35 U.S.C. 102(a) as being disclosed by 

Chakraborty et al. ? (US Patent No. : 6,462,754 Bl), hereinafter Chakraborty. 

A. Chakraborty discloses claim 1, "A method of describing object region data about an 

object in video data including frames [col. 3, Ins. 8-39 and 49-53; col. 5, Ins. 46-67; col. 10, Ins. 

13-14] arranged in a frame advancing direction [see response to arguments below], the method 

comprising: 

extracting an object from each of the frames [col. 7, Ins. 25-59]; 

approximating the object of each of the frames using one of predetermined figures defined by 
representative points [col. 3, Ins. 32-39, wherein predetermined figures correspond to vertices, 
and line such as a spline]; 
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extracting the representative points of the figure for each of the frames, one of the representative 
points in each frame being a reference point represented by a coordinate value and remaining 
representative points in each of the frames being represented by vectors with reference to other 
representative points [col. 4, Ins. 9-15, Ins. 43-46; col. 6, Ins. 36-46; col. 7, In. 25-59; col. 8, In. 
26 - col. 9, In. 67, wherein according to (EQ. 1) x and y would correspond to reference points 
and coordinate values for t defined at first of two consecutive frames]; 
approximating a first trajectory with a first function, the first trajectory being obtained by 
arranging, in the frame advancing direction, position data about the coordinate value of the 
reference point [col. 4, Ins. 50-65; col. 8, In. 26 - col. 9, In. 67, wherein V x (x,y) corresponds to 
"first trajectory with a first function"]; 

approximating second trajectories with second functions, each of the second trajectories being 
obtained by arranging, in the frame advancing direction, the vectors of the remaining 
representative points [wherein V y (x,y) corresponds to "a second trajectory with a second 
function]; and 

describing the object region data using the first and second functions [col. 4, Ins. 50-65; col. 8, 
In. 26 - col. 9, In. 67]" as [detailed]. 

B. Chakraborty discloses claim 2, "The method according to claim 1, wherein said object 
region data comprises information representing a range of frames in which the object exists in 
the video data and information identifying the figure approximating the object region: supra for 
claim 1, wherein [between frames] corresponds to "a range of frames". 

C. Chakraborty discloses claim 3, "The method according to claim 1, wherein said object 
region data comprises one of information representing related information linking to the object 
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and information representing a method of accessing the related information" supra for claim 1 
and furthermore in [col. 3, Ins. 39-56]. 

D. Chakraborty discloses claim 4, "The method according to claim 1, wherein said relative 
position data are components of differential vectors between the one of said the representative 
points and remaining of the representative points" supra for claim 1 and furthermore in [col. 8, 
Ins. 30-39], 

E. Chakraborty discloses claim 5, "The method according to claim 1, wherein said object 
region data comprises parameters of the functions" supra for claims 1 and 4. 

F. Per independent claim 6, this is directed to a method for performing the method of 
independent claim 1, and therefore is rejected to independent claim 1. 

G. Per dependent claims 7-10, these are directed to a method for performing the method of 
dependent claims 2-5, respectively, and therefore are rejected to dependent claims 2-5. 

H. Per independent claims 28, 29 and 34, 35, these are directed to a article of manufacture 
and computer data signal, respectively, for performing the method of independent claims I and 
6, respectively, and therefore are identically rejected to independent claims 1 and 6. 

Claim Rejections - 35 USC §103 
5. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in 
section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are 
such that the subject matter as a whole would have been obvious at the time the invention was made to a person 
having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the 
manner in which the invention was made. 
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6. Claims 1 1-27, 30-32 and 36-38 are rejected under 35 U.S.C. 103(a) as being unpatentable 
over Chakraborty as applied to claim 1-10, 28, 28, 34, 35, above, and further in view of Jasinschi 
et al, (US Patent Number 6,504,569 Bl), hereafter Jasinschi. 

A. Chakraborty discloses claim 1 1, "A method of describing object region data about an 
object in video data over a plurality of frames, said the method comprising: extracting an object 
from each of the frames: approximating the object of each of the frames using one of 
predetermined figures defined by representative points for each of the frames; extracting the 
representative the points of the figure for each of the frames, one of the representative points 
being a reference point represented by a coordinate value and one of the remaining representative 
points being represented by a relative position data with reference to the reference point; 
approximating a first trajectory with a first function, the first trajectory being obtained by 
arranging, in the frames advancing direction, data indicating a position of the reference point; 
and approximating a second trajectory with a second function, the second trajectory being 
obtained by arranging, in the frames advancing direction, the relative position data about the one 
of the remaining points with reference to the reference point; and describing the object region 
data using the first and second functions and depth information of the object" supra for claim 1 
and 6. 

However Chakraborty does not appear to disclose "describing the object region data 

+ 

using depth information of the object", but Jasinschi does in col. 1, Ins. 37-58 at "(10) 
Accordingly the present invention provides a method of generating 2-D extended images from 3- 
D data extracted from a video sequence representing a natural scene. In an image pre-processing 
stage image feature points are determined and subsequently tracked from frame to frame of the 
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video sequence. In a structure-from-motion stage the image feature points are used to estimate 
three-dimensional object velocity and depth. Following these stages depth and motion 
information are post-processed to generate a dense three-dimensional depth map. World 
surfaces, corresponding to extended surfaces, are composed by integrating the three-dimensional 
depth map information."; in col. 2, Ins. 52-65; and col. 3, Ins. 33-36. 

Therefore it would have been obvious to one of ordinary skill in the art at the time the 
invention was made to apply means for interpolating vertices of the objects between frames to 
define motions of the objects of interest so that the objects of interest are tracked disclosed by 
Chakraborty in combination with depth determining information disclosed by Jasinschi, and 
motivated to combine the teachings because it would provide a method of generating 2-D 
extended images from 3-D data extracted from a two-dimensional video sequence as revealed by 
Jasinschi in col. 1, lines 32-34. 

B. Per dependent claims 12 and 13, these are directed to a method for performing the 
method of dependent claims 2 and 3, respectively, and therefore are rejected to claim 1 1 and to 
dependent claims 2 and 3. 

C. Per dependent claim 14, "The method according to claim 1 1, wherein said object region 
data is described by using the depth information of the object and parameters of the functions." is 
disclosed supra by Chakraborty for claim 4 and supra by Jasinschi for claim 1 1 . 

D. Per dependent claim 15, "The method according to claim 1 1, wherein said depth 
information is a relative depth and has a discrete level value" is disclosed supra by Chakraborty 
for claim 4 and supra by Jasinschi for claim 1 1 and in col. 7, Ins. 14-19 at "Step 4: Extract the 
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camera rotation matrix R and the camera translation vector T from the computed essential matrix 
E. Step 5: Given R and T estimate the depth Z.sub.i at every feature point F.sup.i.sub.k. " 
E. Claim 16, "A method of describing object region data about an object in video data over 
a plurality of frames, the method comprising: extracting an object from each of the frames; 
approximating the object of each of the frames using one of predetermined figures defined by 
representative points for each of the frames; extracting the representative points of the figure for 
each of the frames, one of the representative points being a reference point represented by a 
coordinate value and one of the remaining representative points being represented by a relative 
position data with reference to the reference point; approximating a first trajectory with a first 
function, the first trajectory being obtained by arranging, in the frames advancing direction, data 
indicating a position of the reference point; approximating a second trajectory with a second 
function, the second trajectory being obtained by arranging, in the frames advancing direction, 
the relative position data about the one of the remaining points with reference to the reference 
point; and describing the object region data using the first and second functions and display flag 
information indicating a range of frames in which the object or each of the representative points 
is visible or not." is disclosed by Chakraborty supra for claim 1. However Chakraborty does not 
appear to disclose "display flag information indicating a range of frames in which the object or 
each of the representative points is visible or not", but Jasinschi does in col. 4, Ins. 20-28 at "The 
inputs to the 3-D camera parameter estimator 16 are raw video images, denoted by I.sub.k, and 
the corresponding "alpha" images, denoted by A.sub.k. The alpha image is a binary mask that 
determines the "valid" regions inside each image, i.e., the regions of interest or objects, as shown 
in FIG. 3 where FIG. 3 A represents an image I.sub.k from a tennis match and FIG. 3B represents 
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the alpha image A.sub.k for the background object with the tennis player blanked out." Wherein 
[binary mask] corresponds to "display flag information"; and [valid regions] corresponds to 
"object is visible or not". 

Therefore it would have been obvious to one of ordinary skill in the art at the time the 
invention was made to apply object tracked disclosed by Chakraborty in combination with alpha 
images A.sub.k disclosed by Jasinschi, and motivated to combine the teachings because it would 
provide a method of generating 2-D extended images from 3-D data extracted from a two- 
dimensional video sequence as revealed by Jasinschi in col. 1, lines 32-34. 

F. Per dependent claims 17 and 18, these are directed to a method for performing the 
method of dependent claims 2 and 3, respectively, and therefore are rejected to claim 16 and to 
dependent claims 2 and 3. 

G. Per dependent claim 19, "The method according to claim 16, wherein said object region 
data is described by using the display flag information and parameters of the functions" is 
disclosed supra by Chakraborty and Jasinschi for claim 16 supra. Wherein alpha images A.sub.k 
corresponds to display flag information for valid regions. 

H. Claim 20, "A method of describing object region data about an object in video data over 
a plurality of frames, said method comprising: approximating the object using a figure for each 
of said frames; extracting a plurality of points representing the figure for each of said frames; 
approximating trajectories with functions, the trajectories being obtained by arranging, in the 
frames advancing direction, data indicating positions of said plurality of points; and describing 
the object region data using the functions and object passing range information indicating a range 
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where the figure approximating the object exist over said plurality of frames" is disclosed by 
Chakraborty and Jasinschi supra for claims 1 1 and 16. 

However Chakraborty does not appear to disclose "describing the object region data 
using the functions and object passing range information indicating a range where the figure 
approximating the object exist over said plurality of frames.", but Jasinschi does in col. 4, Ins. 
20-28 at "The inputs to the 3-D camera parameter estimator 16 are raw video images, denoted by 
Lsub.k, and the corresponding "alpha" images, denoted by A.sub.k. The alpha image is a binary 
mask that determines the "valid" regions inside each image, i.e., the regions of interest or objects, 
as shown in FIG. 3 where FIG. 3A represents an image I.sub.k from a tennis match and FIG. 3B 
represents the alpha image A.sub.k for the background object with the tennis player blanked 
out " Wherein [object passing range information] corresponds to "sub.k"; and [valid regions] 
corresponds to "object exist". 

Therefore it would have been obvious to one of ordinary skill in the art at the time the 
invention was made to apply object and passing range tracking disclosed by Lee in combination 
ranging information disclosed by Jasinschi, and motivated to combine the teachings because it 
would provide a method of generating 2-D extended images from 3-D data extracted from a two- 
dimensional video sequence as revealed by Jasinschi in col. 1, lines 32-34. 
I. Per dependent claims 21 and 22, these are directed to a method for performing the 
method of dependent claims 2 and 3, respectively, and therefore are rejected to claim 20 and to 
dependent claims 2 and 3. 

J. Per dependent claim 23, "The method according to claim 20, wherein said object region 
data is described by using the object passing range information and parameters of the functions." 
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is disclosed supra by Chakraborty and Jasinschi for claim 20 supra and exemplified by 
Chakraborty. 

K. Per independent claims 30-32 and 36-38, these are directed to a article of manufacture 
and computer data signal, respectively, for performing the method oT independent claims 11, 16, 
and 20, respectively, and therefore are identically rejected to independent claims 11, 16, and 20. 
7. Claims 24-27, 33 and 39 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
Chakraborty et al., (US Patent 6,462,754 Bl), as applied to claims 1-5 above, and further in view 
of "Panoramic Image Mosaics", Heung-Yeung Shum, hereafter Shum. 
A. Claim 24, "A method of describing object region data about an object moving in a 
panorama image formed by combining a plurality of frames with being overlapped, said the 
method comprising: extracting an object from each of the frames: approximating the object of 
each of the frames in the panorama image using one of predetermined figures defined by 
representative points for each of the frames; extracting the representative points of the figure in a 
coordinate system of the panorama image, one of the representative points being a reference 
point represented by a coordinate value and one of the remaining representative points being 
represented by a relative position data with reference to the reference point; approximating a first 
trajectory with a first function, the first trajectory being obtained by arranging, in the frames 
advancing direction, data indicating a position of the reference point; and approximating a 
second trajectory with a second function, the second trajectory being obtained by arranging, in 
the frames advancing direction the relative position data about the one of the remaining points 
with reference to the reference point: and describing the object region data using the first and 
second functions" is disclosed by Chakraborty supra for claim 1. However Chakraborty does not 
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appear to disclose "approximating the object of each of the frames in the panorama image using 
one of predetermined figures defined by representative points for each of the frames; extracting 
the representative points of the figure in a coordinate system of the panorama image, one of the 
representative points being a reference point represented by a coordinate value and one of the 
remaining representative points being represented by a relative position data with reference to 
the reference point", but Shum does in abstract and last paragraph of p. 2, at "This paper presents 
some techniques for constructing panoramic image mosaics from sequences of images. Our 
mosaic representation associates a transformation matrix with each input image, rather than 
explicitly projecting all of the images onto a common surface (e.g., a cylinder). In particular, to 
construct a full view panorama, we introduce a rotational mosaic representation that associates a 
rotation matrix (and optionally a focal length) with each input image. A patch-based alignment 
algorithm is developed to quickly align two images given motion models. Techniques for 
estimating and refining camera focal lengths are also presented. 

In order to reduce accumulated registration errors, we apply global alignment (block adjustment) 
to the whole sequence of images, which results in an optimally registered image mosaic. To 
compensate for small amounts of motion parallax introduced by translations of the camera and 
other unmodeled distortions, we develop a local alignment (deghosting) technique which warps 
each image based on the results of pairwise local image registrations. By combining both global 
and local alignment, we significantly improve the quality of our image mosaics, thereby enabling 
the creation of full view panoramic mosaics with hand-held cameras. 
We also present an inverse texture mapping algorithm for efficiently extracting environment 
maps from our panoramic image mosaics. By mapping the mosaic onto an arbitrary texture- 
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mapped polyhedron surrounding the origin, we can explore the virtual environment using 
standard 3D graphics viewers and hardware without requiring special-purpose players. 
Third, any deviations from the pure parallax-free motion model or ideal pinhole (projective) 
camera model may result in local misregistrations, which are visible as a loss of detail or 
multiple images (ghosting). To overcome this problem, we compute local motion estimates 
(block-based optical flow) between pairs of overlapping images, and use these estimates to warp 
each input image so as to reduce the misregistration. 

Therefore it would have been obvious to one of ordinary skill in the art at the time the 
invention was made to apply object tracking disclosed by Chakraborty in combination with 
panorama image mosaics disclosed by Shum, and motivated to combine the teachings because it 
would provide a technique for constructing panoramic image mosaics from sequences of images 
as disclosed by Shum in abstract. 

B. Per dependent claims 25-27, these are directed to a method for performing the method of 
dependent claims 2, 3, and 5, respectively, and therefore are rejected to claim 24 and to 
dependent claims 2, 3, and 5. 

C. Per independent claim 33 and 39, these are directed to an article of manufacture and 
computer data signal, respectively, for performing the method of independent claim 24 and 
therefore are identically rejected to independent claim 24. 

Response to Arguments 
8. Substance of the first Office Action, mail date 2/22/2005, used in the rejection is 
incorporated herein by reference. 
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Applicant's arguments filed 06/22/2005 have been fully considered but they are not 
persuasive. With respect to the amendments for "frames arranged in a frame advancing 
direction", this corresponds to the inherent normal advancing of frames with respect to time. It 
would be analogous with an analog clock moving in a clockwise direction or even a digital clock 
moving in a clock advancing direction. This aspect is inherent for video frames. 

Applicants have apparently misinterpreted the Chakraborty reference with respect to col. 
8, In. 26 - col. 9, In. 16. Applicants have stated EQ. 1 as: " Ii(x-V x (x,y), y-V y (x,y)) = Ii+i(x,y) 
and that V x and V y relate to summations of partial derivatives of the image intensity at (x, y) that 
are taken over a small neighborhood. Thus, Vx and Vy do not correspond to a trajectory with a 
function'. Stated another way, Chakraborty obtains a trajectory of an object as a whole, but does 
not obtain trajectories over respective representative points of the object." 

However EQ. 1, for two consecutive frames, the motion at each point in the image may 
be described by: 



where I x , I y and I t are the partial derivatives of the image intensity at (x, y) with respect to x, y 
(position) and t (time), respectively. The summations are taken over a small neighborhood 
around the point (x, y). A multi-scale implementation allows for the estimation of large 
motions. Given this, an attempt to find how well the estimated motion may be approximated 
using an affine transformation in block 34. One hypothesis being that if an affine 
approximation is inadequate, it is likely that the interframe motion is large and thus it would be 




the solution for the motion, ((V x (x, y), V y (x, y)) is given as: 



(EQ. 2) 



[see col. 8, Ins. 32-35] 
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appropriate to introduce a node point to subdivide the shot at that instant in time. The shot or 
subshot includes a plurality of frames. The frames include a start frame and end frame and may 
further include intermediate frames. The motion of an object of interest between the start frame 
and the end frame may be defined by points or vertices of the object which have a position. As 
the objects move, their position changes. If the motion is such that further definition of the 
object of interest is needed, node points are added to the intermediate frames to further define 
this motion, for example by using a spline between points or vertices from start to end frames 
through node points in intermediate frames. 

Clearly for (EQ. 1), the intensity function I is with respect to (time) t, not an (index) i. 

Whereby It represents the present intensity at time (t), while It+i represents the next intensity at 

time (t+1) for two consecutives frames and describing the motion at each point (x,y) in the 
image. 

V x (x,y) and V y (x,y) are the velocity functions with respect to motion at each point in the image , 

x x 

wherein the solution for the motion of ((V x (x, y), V y (x, y)) , wherein ( ) corresponds to the 
transpose of the vector is given as (EQ. 2). Thus V x and V y are the functional components of 
vector (EQ. 1): ((V x (x, y), V y (x, y)) and therefore correspond to "trajectory with a function". 

Furthermore since (EQ. 1) describes the motion at each point in the image (x,y), 
Chakreborty essentially describes trajectories [i.e. V x and V y ] over respective points of the 
object [i.e. (x,y)]. Actually the first matrix element of (EQ. 2) given as £ I X 2 should be £ I x . 

Frame reference points are based from start and stop frames, while "representative points 
in each frame being a reference point" correspond to coordinate values (x,y) wherein x and y 
correspond to the coordinate values of motional position for two consecutive frames. 
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Conclusion 

9. Applicant's amendment necessitated the new ground(s) of rejection presented in this 
Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). 
Applicant is reminded of the extension of time policy as set forth in 37 CFR 1. 136(a). 

A shortened statutory period for reply to this final action is set to expire THREE 
MONTHS from the mailing date of this action. In the event a first reply is filed within TWO 
MONTHS of the mailing date of this final action and the advisory action is not mailed until after 
the end of the THREE-MONTH shortened statutory period, then the shortened statutory period 
will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 
CFR 1 .136(a) will be calculated from the mailing date of the advisory action. In no event, 
however, will the statutory period for reply expire later than SIX MONTHS from the date of this 
final action. 

Responses 

10. Responses to this action should be mailed to: Commissioner of Patents and Trademarks, 
Washington, D.C. 20231. 

Inquiries 

1 1 . Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Gregory F. Cunningham whose telephone number is (571) 272- 
7784. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Matthew Bella can be reached on (571) 272-7778. The fax phone number for the 
organization where this application or proceeding is assigned is 703-872-9306. 
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On July IS. 2005 . the Central FAX Number was change to 571-273-8300. This new 
Central FAX Number is the result of relocating the Central FAX server to the Office's 
Alexandria, Virginia campus. 

Most facsimile-transmitted patent application related correspondence is required to be 
sent to the Central FAX Number. To give customers time to adjust to the new Central FAX 
Number, faxes sent to the old number (703-872-9306) will be routed to the new number until 
September 15, 2005. After September 15. 2005 . the old number will no longer be in service and 
571-273-8300 will be the only facsimile number recognized for "centralized delivery". 

Information regarding the status of an application may be obtained from the Patent 
Application Information Retrieval (PAIR) system. Status information for published applications 
may be obtained from either Private PAIR or Public PAIR. Status information for unpublished 
applications is available through Private PAIR only. For more information about the PAIR 
system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR 
system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). 




Gregory F. Cunningham 

Examiner 

Art Unit 2676 
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SUPERVISORY PATENT EXAMINER 
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